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AMENDED APPEAL BRIEF 

Sir: 

This amended brief is in support of the June 12, 2007 notice of appeal of 
the final rejection of the claims. Amendments to the brief have been made in 
response to the Notification of Non-Compliant Appeal Brief mailed on September 
5, 2007. 

(0 Real Party in Interest 

Siemens Corporate Research, Inc., Princeton, N.J. 
(ii) Related Appeals and Interferences 

None. 
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(iii) Status of Claims 

Claims 31-40 are pending in this application. Claims 1-30 have been 
canceled. Claims 31-40 stand rejected. 

The rejection of claims 31-40 are appealed. 

flv) Status of Amendments 

A request for reconsideration after final rejection, submitted on May 3, 
2007 was entered by the Examiner. No amendments have been filed 
subsequent to final rejection. 

(v) Summary of the Claimed Subject Matter 

An augmented reality system allows a user to determine the depth of an 
instrument being inserted into an object from an external perspective. An 
augmented reality view is presented by overlaying a virtual graphics guide onto a 
real view of the object and an instrument. The graphics guide comprises a virtual 
depth marker located outside of the object. The instrument is aligned to the 
graphics guide. The instrument is inserted to a depth determined in the 
augmented view by alignment of a predetermined feature of the instrument with 
the virtual depth marker. The feature is located along the length of the 
instrument at a certain distance from the instrument tip and remains external to 
the object during insertion. 

Two independent claims are involved in the appeal: 
Independent claim 31 is directed to a method for augmented reality 
guided positioning of a real instrument tip within a real target located in a real 
object. A general representation including the method is shown in FIG. 27 of the 
present application, and described in the present specification at p. 55, line 15 - 
p. 56, line 19. The real target located in a real object may, for example, be a 
mass for which a biopsy is desired (p. 52, line 10) located in a patient (p. 52, line 
21). 
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In the ciaimed method, an augmented reality view is presented by 
overlaying a virtual graphics guide onto a real view of the real object and a real 
instrument (p. 16, line 20 - p. 17, line 1). An example of such a guide is guide 
2510 shown in FIG. 25 (p. 54, line 16). The guide includes a virtual depth 
marker, such as the marker 2500 of FIG. 25, located outside the real object (p. 

53, lines 24-26). A real instrument, such as those shown in FIG. 26, is aligned to 
the graphics guide (p. 53, lines 18-22). 

The real instrument is inserted to a depth determined in the augmented 
view by alignment of a predetermined real feature of the real instrument, such as 
features 2610, 2620, 2630, 2640 shown in FIG. 26, with the virtual depth marker 
(p. 51, lines 18-20; p. 53, line 22 - p. 54, line 1). The real feature is located 
along the length of the real instrument at a certain distance from the real 
instrument tip (FIG. 26; p. 51 , lines 23-26). The real feature of the real 
instrument remains external to the real object during insertion (p. 53, line 22 - p. 

54, line 1). 

Independent claim 36 claims an apparatus for augmented reality guided 
instrument positioning of a real instrument tip within a real target located in a real 
object. A generalized apparatus according to the invention, including many 
optional peripheral elements, is shown in FIG. 2. 

The apparatus includes a virtual graphics guide generator and positioner 
280 (FIG. 2; p. 16, lines 2-13) for generating and positioning a virtual graphics 
guide. Illustrative graphics guides are shown in FIGS. 24 and 25. The graphics 
guide includes a virtual depth marker such as marker 2400 of FIG. 24 and 
marker 2500 of FIG. 25 (p. 54, line 6 - p. 55, line 2. The marker is located 
outside of the real object (p. 53, lines 21-22). 

A rendering device 220 (FIG. 2; p. 1 6, line 20 - p. 1 7, line 1 ) is for 
rendering the virtual graphics guide into a real view of the real object and a real 
instrument. The virtual graphics guide is rendered such that the real instrument 
can be inserted to a depth determined in the augmented view by alignment of a 
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predetermined feature of the real instrument with the virtual depth marker (p. 51 , 
lines 18-20; p. 53, line 22 - p. 54, line 1). The real feature, as illustrated by the 
markers 2610, 2620, 2630, 2640 shown in FIG, 26, is located along the length of 
the real instrument at a certain distance from the real instrument tip (p. 51 , lines 
23-26). The real feature of the real instrument remains external to the real object 
during insertion (p. 53, line 22 - p. 54, line 1). 

( vi ) Grounds of rejection to be reviewed on appeal 

(1 ) Obviousness of claims 31-40 as unpatentable over an article 
entitled 'The Expert Surgical Assistant: An Intelligent Virtual 
Environment with Multimodal Input" (Billinghurst) in view of U.S. 
Patent No. 6,470,207 (Simon). 

(vii) Argument 

Billinghurst and Simon Do Not Render 
The Claims Obvious as They Do Not Teach 
All of the Claimed Elements 

Claims 31-40 stand rejected under 35 U.S. C. §103 (a) as obvious in view 
of an article entitled 'The Expert Surgical Assistant; An intelligent Virtual 
Environment with Multimodal Input" (Billinghurst) in view of U.S. Patent No. 
6,470,207 (Simon). 'To establish prima facie obviousness of a claimed 
invention, all the claim limitations must be taught or suggested by the prior art." 
M.P.E.P. §2143.03 (8 th Ed., rev. 4, October 2005), p. 2100-139, citing In re 
Royka, 490 F.2d, 981, 180 U.S.P.Q. 580 (C.C.P.A. 1974) [emphasis in original]. 
The cited references fail to teach all of the limitations in the claims. 

Billinghurst discloses a system for addressing instrument location and 
navigation that tracks the location of the instrument through images. Billinghurst 
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does not teach or disclose "inserting the real instrument to a depth determined in 
the augmented view by alignment of a predetermined real feature of the real 
instrument with the virtual depth marker, the real feature being located along the 
length of the real instrument at a certain distance from the real instrument tip, 
and remains externa! to the real object during insertion" as recited in independent 
claim 31 . Nor does Billinghurst teach or disclose "a rendering device for 
rendering the virtual graphics guide onto a real view of the real object and a real 
instrument such that the real instrument can be inserted to a depth determined in 
the augmented view by alignment of a predetermined real feature of the real 
instrument with the virtual depth marker, the real feature being located along the 
length of the real instrument at a certain distance from the real instrument tip, 
and remains external to the real object during insertion" as recited in independent 
claim 36. 

In the analysis of claims 31 and 36, the office action recites "the graphics 
guide comprising a virtual depth marker located outside of the real object (figure 
2.0, pp. 596)". The referenced image shows a trajectory for instrument 
movement based on a defined starting point. There is no virtual depth marker 
external to the object and Billinghurst does not teach or disclose a real 
instrument having a real feature that is aligned with the virtual depth marker and 
which remains external to the real object (i.e., anatomical structure). The 
instrument tip is marked on the images in Billinghurst with a cross which moves 
according to the user's instrument motion. The CT scan also changes in 
response to instrument depth, so the current images are those corresponding to 
the instrument tip location within the nasal cavity. Billinghurst uses this tracking 
mechanism to monitor the position of the instrument. 

Simon discloses a surgical navigational guidance system which uses one 
or more fluoroscopic x-ray images. Simon uses a virtual cone as a mathematical 
reference to the volume that is being imaged by the X-ray imaging system. The 
location and opening angle of the virtual cone determine the freld-of-view of the 
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X-ray system. Simon determines the coordinates of the virtual cone with respect 
to the coordinate system of the tracking system so that he can overlay a virtual 
model of the tracked instrument spatially correct onto the X-ray image of the 
patient's anatomy. Tracking of the instrument is a prerequisite for Simon's 
method. 

Like Billinghurst, Simon does not teach or disclose the claim elements 
recited in independent claims 31 and 36. The virtual cone is not shown to the 
user and therefore cannot provide any guidance to the user; in particular, it does 
not serve as a virtual depth marker. As indicated in column 12, lines 10-23, the 
virtual cone is known in the coordinate system of the tracking system. The cone 
is used to understand the spatial relationship of the X-ray system's field-of-view 
with the tracked position of the instrument, which allows one to indicate the 
instrument position (known to the system by instrument tracking) in the 
fluoroscopic image with a virtual instrument. The virtual instrument can be used 
by the physician to guide the trajectory of the real instrument. Simon does not 
align a real feature of the instrument with the virtual marker in order to insert the 
instrument to a predetermined depth. 

The office action also refers to column 7, lines 49-51 where Simon 
discloses an instrument embedded with infrared emitters or reflectors. The 
infrared emitters or reflectors are different from the real feature recited in the 
claims because they do not remain external to the object and the infrared 
emitters are used with a tracking sensor (see col, 7, lines 45-61) to communicate 
to a computer the location of the instrument. The emitters would be located on 
the part of the instrument that is inserted into the body to assist in tracking the 
movement of the instrument within the body. 

The combination of Billinghurst and Simon lack all of the claimed elements 
and thus the claims are not obvious in view of this art. Since both Billinghurst 
and Simon fail to teach, disclose or suggest aligning a virtual depth marker with a 
real feature of a real instrument such that the real feature remains external to the 



6 



Appl. No. 09/818,123 

Amended Appeal Brief dated September 21 , 2007 
Reply to Office Action of March 14, 2007 

real object during insertion. Claims 32-35 and 37-40, being dependent upon 
independent claims 31 and 36 respectively, are also not taught, disclosed or 
suggested by the combination of Billinghurst and Simon. The combination 
suggested here is deficient and ineffective as the references, individually and in 
combination, fail to teach all of the claimed elements. Additionally, there is no 
teaching or suggestion in any of the references to modify them in such a manner 
that would result in the claimed invention. 

Conclusion 

Since the references do not teach, suggest or disclose all of the claimed 
elements, the rejection under §103 cannot stand and the claims therefore are 
distinguishable over the cited art. The applicants respectfully request that the 
Board reverses the examiner and direct that the application be passed to 
allowance. 



Respectfully submitted, 

Michele L. Conover 
Reg. No. 34,962 
Attorney for Applicant 

Date: September 21. 2007 

Siemens Corporation 
Intellectual Property Department 
1 70 Wood Avenue South 
Iselin, New Jersey 08830 
(732) 321-3013 
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( vii <) Claims Appendix 

31 . A method for augmented reality guided positioning of a real instrument tip 
within a real target located in a real object comprising the steps of: 

presenting an augmented reality view by overlaying a virtual graphics 
guide onto a real view of the real object and a real instrument, the graphics guide 
comprising a virtual depth marker located outside of the real object; 

aligning the real instrument to the graphics guide; 

inserting the real instrument to a depth determined in the augmented view 
by alignment of a predetermined real feature of the real instrument with the 
virtual depth marker, the real feature being located along the length of the real 
instrument at a certain distance from the real instrument tip, and remains external 
to the real object during insertion. 

32. The method of claim 31 wherein the presenting step further includes 
interactive planning which comprises the steps of: 

determining an optimal location for the real instrument with respect to the 
real target; 

calculating the proximity of the predetermined portion of the real 
instrument to the real target based on the optimal location and the geometry of 
the real, instrument; 

using the proximity calculation to determine the position of the virtual 
depth marker on the virtual graphics guide. 

33. The method of claim 32 wherein the proximity calculation comprises a 
range a proximity measurements. 
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34. The method of claim 32 wherein the proximity calculation corresponds to a 
final forward position of the predetermined portion of the real instrument with 
respect to the real target. 

35. The method of claim 32 wherein the proximity calculation corresponds to a 
distance between the virtual depth marker and a point within the real target. 

36. Apparatus for augmented reality guided instrument positioning of a real 
instrument tip within a real target located in ao real object comprising: 

a virtual graphics guide generator and positioner for generating and 
positioning a virtual graphics guide, the graphics guide comprising a virtual depth 
marker located outside of the real object; and 

a rendering device for rendering the virtual graphics guide onto a real view 
of the real object and a real instrument such that the real instrument can be 
inserted to a depth determined in the augmented view by alignment of a 
predetermined real feature of the real instrument with the virtual depth marker, 
the real feature being located along the length of the real instrument at a certain 
distance from the real instrument tip, and remains external to the real object 
during insertion. 

37. The apparatus of claim 36, wherein said virtual graphics guide generator 
and positioner determines an optimal location for the real instrument with respect 
to the real target, and calculates the proximity of said predetermined portion of 
the real instrument to the real target based on the optimal location and the 
geometry of the real instrument. 

38. The apparatus of claim 37, wherein the proximity comprises a range of 
proximities and said virtual graphics guide generator and positioner determines 
an optimal range of locations for the predetermined portion of the real instrument 
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with respect to the real target, and calculates the range of proximities of the 
predetermined portion of the real instrument to the real target based on the 
optimal range and the geometry of the real instrument. 

39. The apparatus of claim 37, wherein the proximity corresponds to a final 
forward position of the predetermined portion of the real instrument with respect 
to the real target. 

40. The apparatus of claim 36, comprising further a display device to display 
the augmented view rendered by the rendering device to the user. 
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(ix) Evidence Appendix 
Not Applicable. 

( x ) Related Proceedings Appendix 
Not Applicable. 
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